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ABSTRACT 



A computer-automated method of hierarchical event moni- 
toring and analysis within an enterprise network including 
deploying network monitors in the enterprise network, 
detecting, by the network monitors, suspicious network 
activity based on analysis of network traffic data selected 
from the following categories: {network packet data transfer 
commands, network packet data transfer errors, network 
packet data volume, network connection requests, network 
connection denials, error codes included in a network 
packet}, generating, by the monitors, reports of the suspi- 
cious activity, and automatically receiving and integrating 
the reports of suspicious activity, by one or more hierarchi- 
cal monitors. 

22 Claims, 5 Drawing Sheets 

Microfiche Appendix Included 
(10 Microfiche, 952 Pages) 
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HIERARCHICAL EVENT MONITORING 
AND ANALYSIS 

CROSS REFERENCE TO RELATED 
APPLICATION 

This application is a continuation of U.S. application Ser. 
No. 09/188,739 filed Nov. 9, 1998, now U.S. Pat. No. 
6,321,338. 

REFERENCE TO GOVERNMENT FUNDING 

This invention was made with Government support under 
Contract Number F30602-96-C-0294 awarded by DARPA. 
The Government has certain rights in this invention. 

REFERENCE TO APPENDIX 

A microfiche appendix is included as part of the specifi- 
cation. The microfiche includes material subject to copyright 
protection. The copyright owner does not object to the 
facsimile reproduction of the microfiche appendix, as it 
appears in the Patent and Trademark Office patent file or 
records, but otherwise reserves all copyright rights. This 
application contains Microfiche Appendix containing ten 
(10) slides and 956 frames. 

BACKGROUND 

The invention relates to computer networks. 

Computer networks offer users ease and efficiency in 
exchanging information. Networks tend to include conglom- 
erates of integrated commercial and custom-made 
components, interoperating and sharing information at 
increasing levels of demand and capacity. Such varying 
networks manage a growing list of needs including 
transportation, commerce, energy management, 
communications, and defense. 

Unfortunately, the very interoperability and sophisticated 
integration of technology that make networks such valuable 
assets also make them vulnerable to attack, and make 
dependence on networks a potential liability. Numerous 
examples of planned network attacks, such as the Internet 
worm, have shown how interconnectivity can be used to 
spread harmful program code. Accidental outages such as 
the 1980 ARPAnet collapse and the 1990 AT&T collapse 
illustrate how seemingly localized triggering events can 
have globally disastrous effects on widely distributed sys- 
tems. In addition, organized groups have performed mali- 
cious and coordinated attacks against various online targets. 

SUMMARY 

In general, in an aspect, the inventon features a computer- 
automated method of hiererchical event monitoring and 
analysis within and enterprise notwork including deploying 
network monitors in the enterprise notwork, detecting, by 
the network monitors, suspicious network activity based on 
analysis of network traffic data selected from the following 
categories: {network packet data transfer commands, net- 
work packet data trasfer errors, network packet data volume, 
network connection requests, network connection denials, 
error codes included in a network packet}, generating by the 
monitors, reports of the suspicious activity, and automati- 
cally receiving and integrating the reports of suspicious 
ativity, by one or more hierarchical monitors. 

In general, in another aspect, the invention features an 
enterprise network monitoring system including network 
monitors deployed within an enterprise network, the net- 
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work monitors detecting suspicious network activity based 
on analysis of network traffic data selected transfer errors, 
network packet data volume, network connection requests, 
network connection denials, error codes included in a net- 
5 work packet}, the network monitors generating reports of 
the suspisious activity, and one or more hierarchical moni- 
tors in the enterprise network, the hierarchical monitors 
adapted to automatically receive and integrate the reports of 
suspicious activity. 
10 For example, an attack made upon one network entity 
may cause other entities to be alerted. Further, a monitor that 
collects event reports from different monitors may correlate 
activity to identify attacks causing disturbances in more than 
one network entity. 
15 Additionally, statistical analysis of packets handled by a 
virtual private network enable detection of suspicious net- 
work activity despite virtual private network security tech- 
niques such as encryption of the network packets. 
20 Other features and advantages will become apparent from 
the following description, including the drawings, and from 
the claims. 

BRIEF DESCRIPTION OF THE DRAWINGS 

25 FIG. 1 is a diagram of network monitors deployed in an 
enterprise. 

FIG. 2 is a diagram of a network monitor that monitors an 
event stream. 

3Q FIG. 3 is a diagram of a resource object that configures the 
network monitor of FIG. 2. 
FIG. 4 is a flowchart illustrating network surveillance. 
FIG. 5 is a flowchart illustrating multiple short-term 
statistical profiles for comparison against a single long-term 
35 statistical profile. 

FIG. 6 is a diagram of a computer platform suitable for 
deployment of a network monitor. 

DETAILED DESCRIPTION 

40 

Referring to FIG. 1, an enterprise 10 includes different 
domains I2a-12c. Each domain \2a-\2c includes one or 
more computers offering local and network services that 
provide an interface for requests internal and external to the 

45 domain 12a-12c. Network services include features com- 
mon to many network operating systems such as mail, 
HTTP, FTP, remote login, network file systems, finger, 
Kerberos, and SNMP. Some domains 12a-12c may share 
trust relationships with other domains (either peer-to-peer or 

5Q hierarchical). Alternatively, domains 12a-12c may operate 
in complete mistrust of all others, providing outgoing con- 
nections only or severely restricting incoming connections. 
Users may be local to a single domain or may possess 
accounts on multiple domains that allow them to freely 

55 establish connections throughout the enterprise 10. 

As shown, the enterprise 10 includes dynamically 
deployed network monitors 16a-16f that analyze and 
respond to network activity and can interoperate to form an 
analysis hierarchy. The analysis hierarchy provides a frame - 

60 work for the recognition of more global threats to interdo- 
main connectivity, including coordinated attempts to infil- 
trate or destroy connectivity across an entire network 
enterprise 10. The hierarchy includes service monitors 
16a-16c, domain monitors 16rf-16e, and enterprise moni- 

65 tors 16/ 

Service monitors 16a-16c provide local real-time analy- 
sis of network packets (e.g., TCP/IP packets) handled by a 
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network entity 14a-14c. Network entities include gateways, monitor 16fl-16/may propagate to other monitors 16a-16f 

routers, firewalls, or proxy servers. A network entity may throughout the network. Interdomain event analysis is vital 

also be part of a virtual private network. A virtual private to addressing more global, information attacks against the 

network (VPN) is constructed by using public wires to entire enterprise 10. 

connect nodes. For example, a network could use the 5 Referring to FIG. 2, each monitor 16 includes one or more 

Internet ^as the medium ^for transporting data and use ;encryp- ^ en ^ nes 22 ^ ^ ^ cafl ^ 

tion and other security mechanisms to ensure that only , • « jj j j i . j a ¥2 a i 

authorized users access the network and that the data cannot *Y*f*f^ ^ded, deleted, and modified as necessary. In 

, . , » j a , i * u *u the dual-analysis configuration shown, a monitor 16 instan- 

be intercepted. A monitor 16a-l 6/ can analyze packets both . . • * i • j * 

. f a c* a u a c *u i * * tiation includes a signature analysis engine 22 and a statis- 

before and after decryption by a node of the virtual private 1fl . cr • <*a * i 1* 

network profiling engine 24. In general, a momtor 16 may 

¥ - ' . , . • include additional analysis engines that may implement 

Information gathered by a service monitor lfta-lfc can othef forms of aQa] ^ A moni(or lfi ^ includes a 

be disseminated to other monitors 16a-16f, for example, via resolver20 tha t implements a response policy and a resource 

a subscription-based communication scheme. In a Qb - n ^ confi , he monitor 16 ^ monitors 16 

subscnpuon-based scheme client monitors subscribe to 15 mcorporate an application programmers' interface (API) 

receive analysis reports produced by server monitors. As a ^ enhances sulation of monitor and eas< ^ 

monitor 16«-16/ produces analysis reports, the monitor mt ation of third . partv intrusion^letection tools 28, 30. 

16a-l 6/ disseminates these reports asynchronously to sub- . 

scribers. Through subscription, monitors l<w-16/ distrib- Each monitor 16 can ^ event records mat form 311 

uted throughout a large network are able to efficiently 20 eve . nt stream - ^ event stream ma y be denved from a 

disseminate reports of malicious activity without requiring vanet y of sources such 15 TCP/IP network P acket CODlents 

the overhead of synchronous polling. or event records containing analysis reports disseminated by 

Domain monitors Ud-Ue perform surveillance over all ° ther momtors : For example an event record can be formed 

or part of a domain Ua-12c. Domain momtors 16d-16e fr ° m d , ata , ° cluded in ,h , e head f and data se ^ eD \ of a 

correlate intrusion reports disseminated by individual ser- 25 n6tW0 * packet ^ volume °[ f ackets t^™" 1 a ° d 

„• , ,<„ ,<„ .■ „ ,, •„ . . „„ received, however, dictates careful assessment of ways to 

vice monitors loa-loc, providing a domain-wide perspec- , , . , , . „ ' 

»*._.-.,-.../- ... __«r \i jj V . j_- select and organize network packet information into event 

live 01 activity (or patterns of activity). In addition to domain . , " r 

record streams 

surveillance, domain monitors 16«-16c can reconfigure 

system parameters, interface with other monitors beyond a Selection of packets can be based on different criteria, 
domain, and report threats against a domain 12a-12c to 30 Streams of event records can be derived from discarded 
administrators. Domain monitors 16d~\6e can subscribe to traffic ( ie > Packets not allowed through the gateway because 
service-monitors 16a-16c. Where mutual trust among they violate filtering rules), pass-through traffic (i.e., packets 
domains 12a-12c exists, domain monitors 16d-16e may allowed into the internal network from external sources), 
establish peer relationships with one another. Peer-to-peer packets having a common protocol (e.g., all ICMP (Internet 
subscription allows domain monitors 16rf-16e to share 35 Control Message Protocol) packets that reach the gateway), 
analysis reports produced in other domains 12a-12c. packets involving network connection management (e.g., 
Domain monitors 16d-16e may use such reports to dynami- s w » RESET, ACK, [window resize]), and packets targeting 
cally sensitize their local service monitors 16a-16c to mali- P orts t0 wmch an administrator has not assigned any net- 
cious activity found to be occurring outside a domain work service and that also remain unblocked by the firewall. 
12a-12c. Domain monitors 16</-16e may also operate 40 Event streams may also be based on packet source addresses 
within an enterprise hierarchy where they disseminate analy- ( e S> packets whose source addresses match well-known 
sis reports to enterprise monitors 16/ for global correlation. external sites such as satellite offices or have raised suspi- 
Enterprise monitors 16/ correlate activity reports pro- cion from other monitoring efforts) or destination addresses 
duced across the set of monitored domains 12a-12c. Enter- ( e 'S-» P ackets whose destination addresses match a given 
prise 10 surveillance may be used where domains 12a-12c 45 mternal host or workstation). Selection can also implement 
are interconnected under the control of a single organization, application-layer momtormg (e.g., packets targeting a par- 
such as a large privately owned WAN (Wide Area Network). tlcular network service or application). Event records can 
The enterprise 10, however, need not be stable in its con- als0 be P roduced from othe r sources of network packet 
figuration or centrally administered. For example, the enter- information such as report logs produced by network enti- 
prise 10 may exist as an emergent entity through new 50 ties> Event streams can be of verv fine granularity. For 
interconnections of domains 12«-12c. Enterprise 10 surveil- exa mple, a different stream might be derived for commands 
lance is very similar to domain 12a-12c surveillance: an received from different commercial web-browsers since 
enterprise monitor 16/ subscribes to various domain moni- each w eb-browser produces different characteristic network 
tors 16d-16e, just as the domain monitors 16</-16e sub- activity. 

scribed to various service monitors 16a-16c. The enterprise 55 A monitor 16 can also construct interval summary event 

monitor 16/ (or monitors, as it would be important to avoid records, which contain accumulated network traffic statistics 

centralizing any analysis) focuses on net work- wide threats ( e -S-> number of packets and number of kilobytes 

such as Internet worm-like attacks, attacks repeated against transferred). These event records are constructed at the end 

common network services across domains, or coordinated of each interval (e.g., once per N seconds). Event records are 

attacks from multiple domains against a single domain. As 60 forwarded to the analysis engines 22, 24 for analysis, 

an enterprise monitor 16/ recognizes commonalities in intru- The profile engine 22 can use a wide range of multivariate 

sion reports across domains (e.g., the spreading of a worm statistical measures to profile network activity indicated by 

or a mail system attack repeated throughout the enterprise an event stream. A statistical score represents how closely 

10), the monitor 16/ can help domains 12a-12c counter the currently observed usage corresponds to the established 

attack and can sensitize other domains 12a-12c to such 65 patterns of usage. The profiler engine 22 separates profile 

attacks before they are affected. Through correlation and management and the mathematical algorithms used to assess 

sharing of analysis reports, reports of problems found by one the anomaly of events. The profile engine 22 may use a 
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statistical analysis technique described in A. Valdes and D. how long ago the data was collected) values for comparison 

Anderson, "Statistical Methods for Computer Usage to the long-term profile. As a consequence of the aging 

Anomaly Detection Using NIDES", Proceedings of the mechanism, the short-term profile characterizes recent 

Third International Workshop on Rough Sets and Soft activity, where "recent" is determined by a dynamically 

Computing, January 1995, which is incorporated by refer- 5 configurable aging parameters. At update time (typically, a 

ence in its entirety. Such an engine 22 can profile network time of low svstem activity), the update function folds the 

activity via one or more variables called measures. Measures short-term values observed since the last update into the 

can be categorized into four classes: categorical, continuous, long-term profile, and the short-term profile cleared. The 

intensity, and event distribution measures. long-term profile is itself slowly aged to adapt to changes in 

^ . , t _ subject activity. Anomaly scoring compares related 

Categorical measures assume values from a discrete, 10 attributes in the short . term profile aga inst the long-term 

nonordered set of possibilities. Examples of categorical profile. As all evaluations are done against empirical 

measures include network source and destination addresses, distributions, no assumptions of parametric distributions are 

commands (e.g., commands that control data transfer and made, and multi-modal and categorical distributions are 

manage network connections), protocols, error codes (e.g., accommodated. Furthermore, the algorithms require no a 

privilege violations, malformed service requests, and mal- 15 priori knowledge of intrusive or exceptional activity, 

formed packet codes), and port identifiers. The profiler The statistical algorithm adjusts a short-term profile for 

engine 22 can build empirical distributions of the category the measure values observed in the event record. The 

values encountered, even if the list of possible values is distribution of recently observed values is compared against 

open-ended. The engine 22 can have mechanisms for "aging the long-term profile, and a distance between the two is 

out" categories whose long-term probabilities drop below a 20 obtained. The difference is compared to a historically adap- 

threshold. tive deviation. The empirical distribution of this deviation is 

Continuous measures assume values from a continuous or transformed to obtain a score for the event. Anomalous 

ordinal set. Examples include interment time (e.g., differ- events f e *°ff T °!f * ^ ionc ^? a ^ ive 

ence in time stamps between consecutive events from the ^ re threshold t based 0D ^VT™, 80010 

same stream), counting measures such as the number of 25 

c \- , t . j * t i 4 . 4 . makes no assumptions on the modality of the distribution for 

errors or a particular type ooserved in ine recent past, me cont i nu0 us measures. 

volume of data transfers over a period of time, and network r. ^, . , , , . 

, „- / u # i * j i_ c Profiles are provided to the computational engine as 

traffic measures (number of packets and number of dasses defined £ ^ resource object ^ ^ math * matical 

kilobytes). The profiler engine 22 treats continuous, mea- factions for anoma i y scoring , profile maintenance, and 

sures by first allocating bins appropriate to the range of datin do not ^ of the data bei ana . 

values of the underlying measure, and then tracking the lyzed b d what is encoded m the fik dass Eveflt 

frequency of observation of each value range. In this way Election interoperability supports translation of the event 

multi-modal distributions are accommodated and much of stream to lhe file and measun; dasses Al ^ ■ 

the computational machinery used for categorical measures anal sis fof different t of moaitored entities ^ math . 

is shared Contmuous measures are useful not only for ematically similar. This approach imparts great flexibility to 

intrusion detection, but ^also to support the monitoring of the the aQal is in that fadi me constants , u date 

health and status of the network from the perspective of frequencV) measure t and ^ on are tailored to the 

connectivity and throughput. For example, a measure of network emit ^mg monitored. 

traffic volume maintained can detect an abnormal loss in the r™ . , , , , .... 

i i . n - j , . , , C it 40 luG, measure types described above can be used mdi- 

data rate of received packets when this volume falls outside u „^ lin i?„ ,« Z!Lu- #• . a ♦ , . i i * 

h - • rtl „ ^ AA . , vidually or in combination to detect network packet 

historical norms. This sudden drop can be specific both to oH h,' • t • * * c u x. * • *• 

*u i . * j * - t attributes characteristic of intrusion. Such characteristics 

the network entity being monitored and to the tune of day * i j i a . . c / • vuaiatwwuw 

t A tU * j * &z * c i mclude large data transfers (e.g., moving or downloading 

(e.g., the average sustained traffic rate for a major network fi! x _ ; * a _ . > &y . 6 . . .. & 
nrt * • wti , ~j • np f . 1inn . j . . files), an increase m errors (e.g., an increase in privilege 

artery is much different at 11:00 a.m. than at midnight). . . , , • x v . & 

J . ^ ; 45 violations or network packet rejections), network connection 

Intensity measures reflect the intensity of the event stream activity, and abnormal changes in network volume, 

(e.g. , number of ICMP packets) over specified time intervals ^ shown> the monitor 16 also il)dudes a si {UIQ en ^ ne 

(e.g., 1 minute, 10 minutes and 1 hour) Intensity measures 24 ^ si mre en ^ e 24 ^ evem stfeam ^ 

are particularly suited for detecting flooding attacks, while abstract represe ntations of event sequences that are known 

also providing insight into other anomalies. 5Q tQ undesirable acdvity signature-analysis objec- 

Event distribution measures are meta-measures that tives depend on which layer in the hierarchical analysis 
describes how other measures in the profile are affected by scheme the signature engine operates. Service monitor 
each event. For example, an "Is" command in an FTP I6a~16c signature engines 24 attempt to monitor for 
session affects the directory measure, but does not affect attempts to penetrate or interfere with the domain's opera- 
measures related to file transfer. This measure is not inter- 55 tion. The signature engine scans the event stream for events 
esting for all event streams. For example, all network-trafBc that represent attempted exploitations of known attacks 
event records affect the same measures (number of packets against the service, or other activity that stands alone as 
and kilobytes) defined for that event stream, so the event warranting a response from the monitor. Above the service 
distribution does not change. On the other hand, event layer, signature engines 24 scan the aggregate of intrusion 
distribution measures are useful in correlative analysis per- 60 reports from service monitors in an attempt to detect more 
formed by a monitor 16a-16/ that receives reports from global coordinated attack scenarios or scenarios that exploit 
other monitors 16a-16/. interdependences among network services. Layering signa- 

The system maintains and updates a description of behav- ture engine analysis enables the engines 24 to avoid mis- 

ior with respect to these measure types in an updated profile. guided searches along incorrect signature paths in addition 

The profile is subdivided into short-term and long-term 65 to distributing the signature analysis, 

profiles. The short-term profile accumulates values between A signature engines 24 can detect, for example, address 

updates, and exponentially ages (e.g., weighs data based on spoofing, tunneling, source routing, SATAN attacks, and 
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abuse of I CMP messages ("Redirect" and "Destination and correlation of analysis results allows monitors 16^-16/ 

Unreachable" messages in particular). Threshold analysis is to represent and profile global malicious or anomalous 

a rudimentary, inexpensive signature analysis technique that activity that is not visible locally. 

records the occurrence of specific events and, as the name i n addition to external-interface responsibilities, the 

implies, detects when the number of occurrences of that 5 resoiver 20 operates as a fully functional decision engine, 

event surpasses a reasonable count. For example, monitors capable of invoking real-time response measures in response 

can encode thresholds to monitor activity such as the num- t0 malicious or anomalous activity reports produced by the 

ber of fingers, pings, or failed login requests to accounts analysis engines. The resoiver 20 also operates as the center 

such as guest, demo, visitor, anonymous FTP, or employees 0 f imramonitor communication. As the analysis engines 22, 

who have departed the company. 10 24 build intrusion and suspicion reports, they propagate 

Signature engine 24 can also examine the data portion of these reports to the resoiver 20 for further correlation, 

packets in search of a variety of transactions that indicate response, and dissemination to other monitors 16a-16/. The 

suspicious, if not malicious, intentions by an external client. resoiver 20 can also submit runtime configuration requests 

The signature engine 24, for example, can parse FTP traffic to the analysis engines 22, 24, for example, to increase or 

traveling through the firewall or router for unwanted trans- 15 decrease the scope of analyses (e.g., enable or disable 

fers of configuration or specific system data, or anonymous additional signature rules) based on various operating met- 

requests to access non-public portions of the directory rics. These configuration requests could be made as a result 

structure. Similarly, a monitor can analyze anonymous FTP of encountering other intrusion reports from other subscrib- 

sessions to ensure that the file retrievals and uploads/ ers. For example, a report produced by a service monitor 

modifications are limited to specific directories. 20 16a-16c in one domain could be propagated to an enterprise 

Additionally, signature analysis capability can extend to monitor 16/, which in turn sensitizes service monitors in 

session analyses of complex and dangerous, but highly other domains to the same activity, 

useful, services like HTTP or Gopher. The re soiver 20 also operates as the interface mechanism 

Signature analysis can also scan traffic directed at unused between administrators and the monitor 16. From the per- 

ports (i.e., ports to which the administrator has not assigned 25 spective of a resoiver 20, the administrator interface is 

a network service). Here, packet parsing can be used to study simply a subscribing service to which the resoiver 20 may 

network traffic after some threshold volume of traffic, submit reports and receive configuration requests. An 

directed at an unused port, has been exceeded. A signature administrative interface tool can dynamically subscribe and 

engine 24 can also employ a knowledge base of known unsubscribe to any of the deployed resolvers 20, as well as 

telltale packets that are indicative of well-known network- 30 submit configuration requests and asynchronous probes as 

service protocol traffic (e.g., FTP, Telnet, SMTP, HTTP). desired. 

The signature engine 24 then determines whether the monitors \6a-Uf incorporate a bidirectional mes- 
unknown port traffic matches any known packet sets. Such sagmg syste m that uses a standard interface specification for 
comparisons could lead to the discovery of network services communication within and between monitor elements and 
that have been installed without an administrator's knowl- external modules. Using this interface specification, third- 
ec te e - party modules 28, 30 can communicate with monitors. For 
The analysis engines 22, 24 receive large volumes of example, third-party modules 28 can submit event records to 
events and produce smaller volumes of intrusion or suspi- the analysis engines 22, 24 for processing. Additionally, 
cion reports that are then fed to the resoiver 20. The resoiver 4Q third-party modules 30 may also submit and receive analysis 
20 is an expert system that receives the intrusion and results via the resolver's 20 external interfaces. Thus, third- 
suspicion reports produced by the analysis engines 22, 24 party modules 28, 30 can incorporate the results from 
and reports produced externally by other analysis engines to monitors into other surveillance efforts or contribute their 
which it subscribes. Based on these reports, the resoiver 20 results to other monitors 16a-16/. Lastly, the monitor's 16 
invokes responses. Because the volume of intrusion and 45 internal API allows third-party analysis engines to be linked 
suspicion reports is lower than the volume of events directly into the monitor boundary. 

received by the analysis engines 22, 24, the resoiver 20 can Th e message system operates under an asynchronous 

afford the more sophisticated demands of configuration communication model for handling results dissemination 

maintenance and managmg the response handling and exter- a nd processing that is generically referred to as subscription- 

nal interfaces ; necessary for monitor operation. Furthermore, $Q based message passing> Component interoperation is client/ 

the resoiver 20 adds to extensibility by providing the sub- server-based, where a client module may subscribe to 

scnption interface through which third-party analysis tools receive everit data or aDa i ysis resuUs bom servers . 0nce a 

28, 30 can interact and participate in the hierarchical analy- subscription request is accepted by the server, the server 

sis scheme. module forwards events or analysis results to the client 

Upon its initialization, the resoiver 20 initiates authenti- 55 automatically as data becomes available, and may dynami- 

cation and subscription sessions with those monitors cally reconfigure itself as requested by the client's control 

16a-16/ whose identities appear in the monitor's 16 requests. This asynchronous model reduces the need for 

subscription-list (46 FIG. 3). The resoiver 20 also handles all client probes and acknowledgments, 

incoming requests by subscribers, which must authenticate nc interface supports an implementation-neutral com- 

themselves to the resoiver 20. Once a subscription session is 60 mun ication framework that separates the programmer's 

established with a subscriber monitor, the resoiver 20 acts as interface specification and the issues of message transport, 

the primary interface through which configuration requests The interface specification embodies no assumptions about 

are received and intrusion reports are disseminated. implementation languages, host platform, or a network. The 

Thus, resolvers 20 can request and receive reports from transport layer is architecturally isolated from the internals 

other resolvers at lower layers in the analysis hierarchy. The 65 of the monitors so that transport modules may be readily 

resoiver 20 forwards analysis reports received from sub- introduced and replaced as protocols and security require - 

scribees to the analysis engines 22, 24. This tiered collection ments are negotiated between module developers. The 
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interface, specification involves the definition of the mes- monitor transport modules to address security and reliability 
sages that the various intrusion-detection modules must issues differently than how the intermonitor transport mod- 
convey to one another and how these messages should be ules address security and reliability. While intramonitor 
processed. The message structure and content are specified communication may more commonly involve interprocess 
in a completely implementation-neutral context. 5 communication within a single host, intermonitor commu- 

Both intramonitor and intermonitor communication nication will most commonly involve cross-platform net- 
employ identical subscription-based client-server models. worked interoperation. For example, the intramonitor trans- 
With respect to intermonitor communication, the resolver 20 P ort mechanisms may employ unnamed pipes which 
operates as a client to the analysis engines, and the analysis provides a kernel-enforced private interprocess communi- 
engines 22, 24 operate as clients to the event filters. Through *° cation channel between the monitor 16 components (this 
the internal message system, the resolver 20 submits con- assumes a process hierarchy within the monitor 16 
figuration requests to the analysis engines 22, 24, and architecture). The monitor's 16 external transport, however, 
receives from the analysis engines 22, 24 their analysis W M more likely export data through untrusted network 
results. The analysis engines 22, 24 operate as servers connections and thus require more extensive security man- 
providing the resolver 20 with intrusion or suspicion reports 35 agement. To ensure the security and integrity of the message 
either asynchronously or upon request. Similarly, the analy- exchange, the external transport may employ public/private 
sis engines 22, 24 are responsible for establishing and kev authentication protocols and session key exchange, 
maintaining a communication link with an event collection Using this same interface, third-party analysis tools may 
method (or event filter) and prompting the reconfiguration of authenticate and exchange analysis results and configuration 
the collection method's filtering semantics when necessary. 20 information in a well-defined, secure manner. 

Intermonitor communication also operates using the The pluggable transport permits flexibility in negotiating 

subscription-based hierarchy. A domain monitor 16d-16e security features and protocol usage with third parties, 

subscribes to the analysis results produced by service moni- Incorporation of a commercially available network manage- 

tors 16a-16c, and then propagates its own analytical reports ment system can deliver monitoring results relating to 

to its parent enterprise monitor 16/. The enterprise monitor 25 security, reliability, availability, performance, and other 

16/ operates as a client to one or more domain monitors attributes. The network management system may in turn 

16d-16e, allowing them to correlate and model enterprise- subscribe to monitor produced results in order to influence 

wide activity from the domain-layer results. Domain moni- network reconfiguration. 

tors 16rf-16e operate as servers to the enterprise monitors All monitors (service, domain, and enterprise) 16a-16/ 
16/, and as clients to the service monitors 16a-16c deployed 30 use the same monitor code-base. However, monitors may 
throughout their domain 12a-12c. This message scheme can include different resource objects 32 having different con- 
operate substantially the same if correlation were to continue figuration data and methods. This reusable software archi- 
at higher layers of abstraction beyond enterprise 10 analysis. tecture can reduce implementation and maintenance efforts. 

Intramonitor and intermonitor programming interfaces 35 Customizing and dynamically configuring a monitor 16 thus 

are substantially the same. These interfaces can be subdi- becomes a question of building and/or modifying the 

vided into five categories of interoperation: channel initial- resource object 32. 

ization and termination, channel synchronization, dynamic Referring to FIG. 3, the resource object 32 contains the 

configuration, server probing, and report/event dissemina- operating parameters for each of the monitor's 16 compo- 

tion. Clients are responsible for initiating and terminating ^ nents as well as the analysis semantics (e.g., the profiler 

channel sessions with servers. Clients are also responsible engine's 22 measure and category definition, or the signa- 

for managing channel synchronization in the event of errors ture engine's 24 penetration rule-base) necessary to process 

in message sequencing or periods of failed or slow response an event stream. After defining a resource object 32 to 

(i.e., "I'm alive" confirmations). Clients may also submit implement a particular set of analyses on an event stream, 

dynamic configuration requests to servers. For example, an 45 the resource object 32 may be reused by other monitors 16 

analysis engine 22, 24 may request an event collection deployed to analyze equivalent event streams. For example, 

method to modify its filtering semantics. Clients may also the resource object 32 for a domain's router may be reused 

probe servers for report summaries or additional event as other monitors 16 are deployed for other routers in a 

information. Lastly, servers may send clients intrusion/ domain 12a-12c. A library of resource objects 32 provides 

suspicion reports in response to client probes or in an 5Q prefabricated resource objects 32 for commonly available 

asynchronous dissemination mode. network entities. 

The second part of the message system framework The resource object 32 provides a pluggable configuration 

involves specification of a transport mechanism used to module for tuning the generic monitor code-base to a 

establish a given communication channel between monitors specific event stream. The resource object 32 includes 

16a-16/ or possibly between a monitor I60-I6/ and a 55 configurable event structures 34, analysis unit configuration 

third-party security module. All implementation dependen- 38a-38n, engine configuration 40a-40/i, resolver configu- 

cies within the message system framework are addressed by ration 42, decision unit configuration 44, subscription list 

pluggable transport modules. Transport modules are specific data 46, and response methods 48. 

to the participating intrusion-detection modules, their Configurable event structures 34 define the structure of 

respective hosts, and potentially to the network— should the 60 evem records and analysis result records. The monitor 

modules require cross-platform interoperation. Instantiating code-base maintains no internal dependence on the content 

a monitor 16a-16/may involve incorporation of the neces- 0 r format of any given event stream or the analysis results 

sary transport module(s) (for both internal and external produced from analyzing the event stream. Rather, the 

communication). resource object 32 provides a universally applicable syntax 

The transport modules that handle intramonitor commu- 65 for specifying the structure of event records and analysis 

nication may be different from the transport modules that results. Event records are defined based on the contents of an 

handle intermonitor communication. This allows the intra- event stream(s). Analysis result structures are used to pack- 
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age the findings produced by analysis engines. Event records potential stability problem if dynamic modifications are not 

and analysis results are defined similarly to allow the tightly restricted to avoid cyclic modifications. To address 

eventual hierarchical processing of analysis results as event this issue, monitors 16 can be configured to accept configu- 

records by subscriber monitors. ration requests from only higher-level monitors 16. 

Event-collection methods 36 gather and parse event 5 Referring to FIG. 4, a monitor performs network surveil- 

records for analysis engine processing. Processing by analy- lance bv monitoring 66 a stream of network packets. The 

sis engines is controlled by engine configuration 40*^0* momtor bu , llds * stat f lcal mo ^ el ° f ° et ™? rk from 

variables and data structures That specify the operating th * \ nei ™* ****** *> r f am P e > ^ bmkhn S 68 lon f term 

configuration of a fielded monitor's analysis engine^). The * nd short-*™ f ^ cal Pi? files from mea *f * denved 

- 4 - * H r in from the network packets. Ine measures include measures 

resource object 32 maintains a separate collection of oper- 10 ~, , ^ , , , ^ 

4 . t c . , * . • 4 * • * j * ii that can show anomalous network activity characteristic of 

ating parameters for each analysis engine instantiated in the , . 4 , / j ^ 

J~- f ;_ 1/: A „„ K ■ - # °- aCn • i j network intrusion such as measures that describe data 

monitor 16. Analysis unit configuration ooa-oon include - . . M 

r t - Li t t * j c ti_ i j transfers, network connections, privilege and network 

configuration variables that define the semantics employed * . , . . . ' F , & ~ Z~ . 

u.. *uZ i lf „„ ^ ^ tU t 4 „ errors, and abnormal levels of network traffic. The monitor 

by the analysis engine to process the event stream. „ n . , . - t 

' ; * : . 1<; can compare 70 the long-term and short-term profiles to 

TTieresolver configuration 42 mcludes operating param- detect suspicious network activity . B ased on this 

eters that specify the configuraUon of the resolver's mternal com p aris0 n, the monitor can respond 72 by reporting the 

modules. The decision unit configuration 44 describes activity t0 another monitor or b executing a countermea- 

semantics used by the resolver s decision unit for merging sure response . More information can be found in P. Porras 

the analysis results from the various analysis engines. The and A> Valdes « Live Xraffic ^ ^ of Tcp/I p Galeway5 « 

semantics include the response cntena used to invoke coun- Net works and Distributed Systems Security Symposium, 

termeasure handler^ A resource object 32 may also include March 1998> which ^ by reference in its 

response methods 48. Response methods 48 include prepro- entirety 

grammed countermeasure methods that the resolver may A r , .,, A 4 iL . iL , £ ^ , 

r , f . i jl . « A few examples can illustrate this method of network 

mvoke as event records are received. A response method 48 xr . , . c . . . 

„ i *■ c a* • 25 surveillance. Network intrusion frequently causes large data 

includes evaluation metrics for determining the circum- « r r i i. • , i i . , j 

u,u ,l j u i j u i i t*u transfers, for example, when an intruder seeks to download 

stances under which the method should be invoked. These r.i„,. nr ™i « ♦ n **u u r i u *• 

m , • • i i 4 . , t , . • . u * i 4 4 « sensitive files or replace system files with harmful substi- 

metnes include a threshold metric that corresponds to the H<# „ A «i « j ♦ ♦ i j.* * 

mflnril „ , j j u *u «i ■ tutes. A statistical profile to detect anomalous data transfers 

measure values and scores produced by the profiler engine . Ut . , * 4 . r C1 4 - 

« , . 4 . 4 , \ j * l r iL might include a contmuous measure of file transfer size, a 

22 and seventy metrics that correspond to subsets of the mo „„„ Aft L opn „„ fl _ , f - # . 4 ' - 

* . j ** i » /j j . ■ . • 30 categoncal measure of the source or destination directory of 

associated attack sequences defined within the resource tUa a ? a • . c a 

ob'ect 32 transfer, and an intensity measure of commands 

J corresponding to data transfers (e.g., commands that down- 

Countermeasures range from very passive responses, such i oad data ). Th ese mea sures can detect a wide variety of data 

as report dissemination to other monitors 16a-16/ or transfer techniques such as a large volume of small data 

administrators, to highly aggressive actions, such as sever- 35 transfers via e-mail or downloading large files en masse. The 

ing a communication channel or the reconfiguration of monitor may distinguish between network packets based on 

logging facilities within network components (e.g., routers, the time such packets were received by the network entity, 

firewalls, network services, audit daemons). An active permitting statistical analysis to distinguish between a nor- 

response may invoke handlers that validate the integrity of mal data iT3in ster during a workday and an abnormal data 

network services or other assets to ensure that privileged 4Q transfer on a weekend evening. 

network services have not been subverted Monitors Attempted network intrusion may also produce anoma- 

16a-16/may invoke probes in an attempt to gather as much lous levels of errors For } cate gorical and intensity 

countenntelhgence about the source of suspicious traffic by measures derived from ^ ^ 

using features such as traceroute or finger. aUempts t0 access protecled files> 6 directorieSj 0 / other net . 

The resource object 32 may include a subscription list 46 45 work assets. Of course, privilege errors occur during normal 
that includes information necessary for establishing network operation as users mistype commands or attempt to 
subscription-based communication sessions, which may perform an operation unknowingly prohibited. By compar- 
include network address information and public keys used ing the long-term and short-term statistical profiles, a moni- 
by the monitor to authenticate potential clients and servers. tor can distinguish between normal error levels and levels 
The subscription list 46 enables transmission or reception of 50 indicative of intrusion without burdening a network admin- 
messages that report malicious or anomalous activity istrator with the task of arbitrarily setting an unvarying 
between monitors. The most obvious examples where rela- threshold. Other measures based on errors, such as codes 
tionships are important involve interdependencies among describing why a network entity rejected a network packet 
network services that make local policy decisions. For enable a monitor to detect attempts to infiltrate a network 
example, the interdependencies between access checks per- 55 with suspicious packets. 

formed during network file system mounting and the IP Attempted network intrusion can also be detected by 

mapping of the DNS service. An unexpected mount moni- measures derived from network connection information For 

tored by the network file system service may be responded example, a measure may be formed from the correlation 

to differently if the DNS monitor informs the network file ( e . g ., a ratio or a difference) of the number of SYN connec- 

system monitor of suspicious updates to the mount request- 60 tfon request messages with the number of SYN_ACK 

or s DNS mapping. connection acknowledgment messages and/or the number of 

The contents of the resource object 32 are defined and ICMP messages sent. Generally, SYN requests received 

utilized during momtor 16 initialization. In addition, these should balance with respect to the total of SYN_ACK and 

fields may be modified by internal monitor 16 components, ICMP messages sent. That is, flow into and out-of a network 

and by authorized external clients using the monitor's 16 65 entity should be conserved. An imbalance can indicate 

API. Modifying the resource object 32 permits adaptive repeated unsuccessful attempts to connect with a system, 

analysis of an event stream, however, it also introduces a perhaps corresponding to a methodical search for an entry 



04/05/2004, EAST Version: 1.4.1 



US 6,484,203 Bl 

13 14 

point to a system. Alternatively, intensity measures of Referring to FIG. 6, a computer platform 14 suitable for 

transport-layer connection requests, such as a volume analy- executing a network monitor 16 includes a display 50, a 

sis of SYN-RST messages, could indicate the occurrence of keyboard 54, a pointing device 58 such as a mouse, and a 

a SYN-attack against port availability or possibly port- digital computer 56. The digital computer 56 includes 

scanning. Variants of this can include intensity measures of 5 memory 62, a processor 60, a mass storage device 64a, and 

TCP/FIN messages, considered a more stealthy form of port other customary components such as a memory bus and 

scanning. peripheral bus. The platform 14 may further include a 

iL * i** • ¥"■• network connection 52. 

Many other measures can detect network intrusion. For . . , . ^ . . , - 

■ « j i i. *.!•»* *• • * c * *• 11 Mass storage device 64a can store instructions that form 

example, "doorknob rattling, testing a variety of potentially a mstmclions be lransferred to memory 

valid commands to gain access (e.g trying to access a 10 62 and processor 60 in the course of operation. The instruc- 

system account with a password of system ), can be lions 16 can cause the dis { 50 tQ di j ^ Wa m 

detected by a variety of categorical measures. A categorical mter face such as a graphical user interface. Of course, 

measure of commands included in network packets can instructions may be stored on a variety of mass storage 

identify an unusual short-term set of commands indicative of devices such as a floppy disk 64b, CD-ROM 64c, or PROM 

"doorknob-rattling." Similarly, a categorical measure of 15 (not shown). 

protocol requests may also detect an unlikely mix of such other embodiments are within the scope of the following 

requests. claims. 

Measures of network packet volume can also help detect What is claimed is: 
malicious traffic, such as traffic intended to cause service 1- A computer-automated method of hierarchical event 
denials or perform intelligence gathering, where such traffic 20 monitoring and analysis within an enterprise network corn- 
may not necessarily be violating filtering policies. A mea- prising: 

sure reflecting a sharp increase in the overall volume of deploying a plurality of network monitors in the enter- 
discarded packets as well as a measure analyzing the dis- prise network; 

position of the discarded packets can provide insight into detecting, by the network monitors, suspicious network 

unintentionally malformed packets resulting from poor line 25 activity based on analysis of network traffic data 

quality or internal errors in neighboring hosts. High volumes selected from the following categories: {network 

of discarded packets can also indicate more maliciously P acket data tra nsfer commands, network packet data 

intended transmissions such as scanning of UPD ports or IP transfer errors, network packet data volume, network 

address scanning via ICMP echoes. Excessive number of connection requests, network connection denials, error 

mail expansion request commands (EXPN) may indicate 30 ^ incl u ud ^ m a network P acke * . 

intelligence gathering, for example, by spammers. ^activit^'and m ° mt0TS ' rep ° rtS ° f SMd sus P icious 

A long-term and short-term statistical profile can be . *■ 'n * • j ♦ . t . lL - 
4 j r . , , ™_ / , automatically receiving and integrating the reports of 
generated for each event stream. Thus, different event „, . . if u- u- i 
* „ „ , * * j * * j * £c . suspicious activity, by one or more hierarchical mom- 
streams can "slice" network packet data in different ways. 35 torg 3 J 

For example, an event stream may select only network <> ti, ' *u a e \ ■ * u ■** 

, . , . J « . „. t 2. The method of claim 1, wherein integrating comprises 

packets having a source address corresponding to a satellite , 4 - ■ t . ^ a j i • 

office. Tfcus, a long-term and short-term profile will be ™£s " & Myitlg t^"' 

generated for the particular satellite office. Thus, although a a ^. *u j r i • * l ■ • . c ,i. 

satellite office ma have more rivile es and sh Id b method or claim 1, wherein integrating further 

^j*! 06 aVC . more P nvi e &f s an s ou G 40 comprises invoking countermeasures to a suspected attack, 

expected to use more system resources than other external a ^n. *u a c i • i i_ • i t-. * 

AA <ii * ♦ ir. «r j , ,„ 4. The method of claim 1, wherein the plurality of 

addresses, a profile of satellite office use can detect address t - , , A r» ¥ c i ■ r . 

c » r a c - i . • c i network monitors include an API for encapsulation of mom- 
spoofing (i.e., modifying packet information to have a _ A , . . 4 . . ... , j 4 t 

a a c *u . fi* az \ tor functions and integration of third-party tools, 

source address of the satellite office). - ™ , c , . - . . X \ . , 

7 5. Ine method of claim 1, wherein the enterprise network 

The same network packet event may produce records in 45 ^ a TCP/IP network, 

more than one event stream. For example, one event stream 6 . method of claim 1? wherein the network monitors 

may monitor packets for FTP commands while another are dep l 0 yed at one or more of the following facilities of the 

event stream monitors packets from a particular address. In enterprise network: {gateways, routers, proxy servers}, 

this case, an FTP command from the address would produce 7 . me thod of claim 1, wherein deploying the network 

an event record in each stream. 50 mon itors includes placing a plurality of service monitors 

Referring to FIG. 5, a monitor may also "deinterleave." among multiple domains of the enterprise network. 

That is, the monitor may create and update 74, 76 more than 8. The method of claim 7, wherein receiving and inte- 

one short-term profile for comparison 78 against a single grating is performed by a domain monitor with respect to a 

long-term profile by identifying one of the multiple short- plurality of service monitors within the domain monitor's 

term profiles that will be updated by an event record in an 55 associated network domain. 

event stream. For example, at any one time a network entity 9. The method of claim 1, wherein deploying the network 

may handle several FTP "anonymous" sessions. If each monitors includes deploying a plurality of domain monitors 

network packet for all anonymous sessions were placed in a within the enterprise network, each domain monitor being 

single short-term statistical profile, potentially intrusive associated with a corresponding domain of the enterprise 

activity of one anonymous session may be statistically 60 network. 

ameliorated by non-intrusive sessions. By creating and 10. The method of claim 9, wherein receiving and inte- 

updating short-term statistical profiles for each anonymous grating is performed by an enterprise monitor with respect to 

session, each anonymous session can be compared against a plurality of domain monitors within the enterprise net- 

the long-term profile of a normal FTP anonymous session. work. 

Deinterleaving can be done for a variety of sessions includ- 65 11. The method of claim 9, wherein the plurality of 

ing HTTP sessions (e.g., a short-term profile for each domain monitors within the enterprise network establish 

browser session). peer-to-peer relationships with one another. 
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12. An enterprise network monitoring system comprising: 16. The system of claim 12, wherein the enterprise 
a plurality of network monitors deployed within an enter- network is a TCP/IP network. 

prise network, said plurality of network monitors 17. The system of claim 12, wherein the network monitors 

detecting suspicious network activity based on analysis are ^Ployed at one or more of the following facilities of the 

of network traffic data selected from the following 5 «««Fn» nelwork: {gateways routers, proxy servers}, 

categories: {network packet data transfer commands, , 18 s 5» tem . of 1 c ! aun U > w ^ Kl ° plurality of 

network oacket data transfer errors network oacket nelwork monitors includes a plurality of service monitors 

ne work packet data transrer errors, network packet affl multiple domams of , he enterprise network, 

data volume, ne work connection requests, network „ ^ s £ tem of daim lg , domain moni(or 

C acS-° n erl ° r C 65 ,D m * 10 associated with the plurality of service monitors within the 

P '* domain monitor's associated network domain is adapted to 

said network monitors generating reports of said suspi- automatically receive and integrate the reports of suspicious 

cious activity; and activity, 

one or more hierarchical monitors in the enterprise 20. The system of claim 12, wherein the plurality of 

network, the hierarchical monitors adapted to automati- 35 network monitors include a plurality of domain monitors 

cally receive and integrate the reports of suspicious within the enterprise network, each domain monitor being 

activity. associated with a corresponding domain of the enterprise 

13. The system of claim 12, wherein the integration network. 

comprises correlating intrusion reports reflecting underlying 21. The system of claim 20, wherein an enterprise monitor 

commonalities. 2Q associated with a plurality of domain monitors is adapted to 

14. The system of claim 12, wherein the integration automatically receive and integrate the reports of suspicious 
further comprises invoking countermeasures to a suspected activity. 

attack. 22. The system of claim 20, wherein the plurality of 

15. The system of claim 12, wherein the plurality of domain monitors within the enterprise network interface as 
network monitors include an application programming inter- 25 a plurality of peer-to-peer relationships with one another, 
face (API) for encapsulation of monitor functions and inte- 
gration of third-party tools. ***** 6 
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